I would say plugin is probably best - at least as a starting point - as it would benefit from the plugin update framework.

+1 on the plugin route for delivering the functionality to users.

Ignoring service chaining (?), presumably you could create a test composition per Spin component? That might even be ideal if you could compose multiple components as you wouldn't have to build anything you aren't currently testing.

But which test composition do we end up actually calling? The spin-test host calls the run function to kick off the test. This run function is exported by the user-written test runner component. This test runner component is ultimately what calls the router (one or more times) as part of the test. If we create N test compositions, N - 1 of those compositions are not meant to receive that request (and calling their run function would just fail). At the core of the issue is that we don't really known which component needs to called before hand - the router is what makes that decision, but the router cannot compose with an arbitrary number of components to route to.

The only way I can see see to work around this are the two proposed solutions: moving routing into the host or synthesizing a router component.

Ohhh I was thinking that the developer would explicitly map tests to components, but you are looking at doing that via regular URL path routing. Yeah I would think in the short term that would require the host to do the routing.

I think this is a scenario that would become (at least for me) much clearer with an example of the (proposed/desired/currently possible/default) developer experience for writing tests. I think there are a few decisions that might impact the decisions here:

• does each Spin component come with its own sets of tests, or is there a single test "project" for the entire Spin application?
• depending on ^^, there could be a scenario where the test is linked to a single component, which means the question is no longer difficult to answer.

You could also (statically/macro-ly) generate a family of routers (router1, router2, ...). Annoying, but probably fine as a medium-term hack.

This wouldn't work either (without changing the host to be smarter) as something needs to split one request into a call to one of N possible components. Composing N components with 1 component (where N is not known ahead of time) is not possible.

What about 1 router with many identical component instance imports? It would be a little fiddly but you should be able to derive one consistent ordering of spin components from a manifest which would let the composition step fill the imports in the same order that the router then expects them. You would also need some way to fill all the unused import slots, either a static dummy component that always traps or some Linker jiggery-pokery.

So you're suggesting a single router component with some arbitrary amount of imports that components can plug into and we either plug those with actual Spin app components or with a dummy trapping component once we run out of Spin app components?

Yep

This feels like something that may become dramatically easier with CM async (to the extent that magically fixes all our nested composition woes).

Hmm... I'm not sure I follow - none of the triggers (with the exception of wasi:http/incoming-handler which we already support) are necessarily built to try to accommodate async. I don't see how CM async improves the situation here. Could you explain more about what you mean?

Ah yeah sorry the train of consciousness left the station early there...

I think each trigger type could have a test driver adapter to wrap the component-under-test and test driver to implement the runner world; this is impossible today because the host has to provide resource factories to glue everything together.

I suppose we could also do both (eventually), with a compatible schema reused between spin.toml [component.<id>.tool] for one-test-per-component and tests/spin-test.toml for more complex scenarios.

Another option to consider might be the application.tool section in spin.toml, e.g. [application.tool.spin-test.test-a]. I'm not sure I would actually like that but it would avoid having another config file.

This refers to "a single component" from the perspective of the Spin application manifest I assume — if my Spin component build process generates a statically linked component, I would expect testing to continue to work as expected.

Is that accurate?

As I was going through the test examples from the implementation repository, I was thinking about the wasi:cli/imports imports — will this contain a programmatic API to set the behavior of all WASI functionality?
Will there be a subset available to begin with?

I'm curious what the API through which we expose that to users in tests will look like.

The example test I ran from the implementation repo did log to stdout -- is this already supported?

nit: leftover >

nit: formatting for WIT source and comments seems off in parts of this file.

+1 on the plugin route for delivering the functionality to users.

I think this is a scenario that would become (at least for me) much clearer with an example of the (proposed/desired/currently possible/default) developer experience for writing tests. I think there are a few decisions that might impact the decisions here:

• does each Spin component come with its own sets of tests, or is there a single test "project" for the entire Spin application?
• depending on ^^, there could be a scenario where the test is linked to a single component, which means the question is no longer difficult to answer.